to: @jacknagz
cc: @airbnb/streamalert-maintainers
size: small
resolves N/A

Background

Discovered an issue with nested type conversion where the schema would still pass even if a value of a nested key was not of the right type.

1. Consider, for instance, you have this log defined in logs.json:

{
  "test:log_name": {
    "schema": {
      "key_01": "string",
      "key_02": "string"
    },
    "parser": "json",
    "configuration": {
      "json_path": "results[*]",
      "envelope_keys": {
        "host": "string",
        "ip": "string"
    }
  }
}

2. And the incoming log looks like this:

{
  "host": "host_name",
  "ip": "0.0.0.0",
  "results": [
    {
      "key_01": "value_01",
      "key_02": "value_02"
    }
  ]
}

3. When streamalert:envelope_keys is used and this log goes through classifying, the resulting log will look something like:

{
  "key01": "value_01",
  "key02": "value_02",
  "streamalert:envelope_keys": {
    "host": "host_name",
    "ip": "0.0.0.0"
  }
}

4. And the schema that gets constructed during parsing this log will look something like this:

{
  "key01": "string",
  "key02": "string",
  "streamalert:envelope_keys": {
    "host": "string",
    "ip": "string"
  }
}

In the above schema definition (# 1) if you change the "host" key's value within "envelope_keys" to "integer", the parsing of this log will succeed, but will log the following error:

StreamAlert [ERROR]: Invalid schema. Value for key [host] is not an int: host_name

Changes

• This change ensure that all value types defined will fail the parsing if the value does not match by returning the recursive result of StreamClassifier._convert_type from within the method.

Testing

Adding unit test to validate that a failed type conversion of a nested value will result in a classification failure.